This pull request ports the Duplicate Detection framework from DSpace CRIS 7 into DSpace.

Attribution: Ported from DSpace-CRIS and extended by The Library Code with support of Technische Universität Berlin

DSpace Backend PR: DSpace/DSpace#8415
REST Contract PR: TBD

One outstanding issue
For a new submission, the default behaviour when no matches are found should be to keep the section hidden, but instead, the default message indicates that decisions are all registered.
Fixed, see comment #1732 (comment)

To test

• Make sure both the dspace-angular and DSpace PRs are applied
• Enable the ‘dedup’ event consumer in dspace.cfg and add the ‘detect-duplicate’ step to a submission process in item-submission.xml
• For best results, configure autosave on the dc.title field and any others you’re planning to test (eg. dc.identifier.doi, dc.identifier.isbn or so on). This means that whenever a change is saved to dc.title, the duplicate detection section will update with a new list of potential matches
• Submit an item to a collection with the detect-duplicate step enabled. Enter some text in the title so that

Notes for PR reviewers

• There is a new ‘dedup’ table in the database, this should be added by flyway
• There is a new ‘dedup’ Solr core
• The ‘dedup’ consumer is not in the default event consumers by default, so it’ll need to be added for practical testing
• The fuzzy search signature is commented out of the deduplication.xml spring config by default, so uncomment that to test fuzzy matches (the default MD5 signatures for exact metadata matches still work)

Notes for 4Science devs (regarding changes from CRIS)

• SearchDeduplication was renamed to DeduplicationPluginService as that seemed more fitting to what it actually does
• tmpMapFilter, tmpFilter, other similar names of maps and lists were renamed to suit their purpose (eg. signatureSearchFilters)
• A lot of unused methods, variables, classes were stripped out during the port to keep code lean and easier to review. If some were partial implementations of future work, we can bring them in with a subsequent PR

As I’ve added an authorization check (configurable, enabled by default) so that the user needs read access to each item in the potential matches list, I manually disabled it for some integration tests and leave it for others, and explicitly test it in a new integration test

I’ve added javadoc and most is pretty straightforward but please correct me if I misunderstood terms such as ‘fake', ‘reader’

Checklist

This checklist provides a reminder of what we are going to look for when reviewing your PR. You need not complete this checklist prior to creating your PR (draft PRs are always welcome). If you are unsure about an item in the checklist, don't hesitate to ask. We're here to help!

• My PR is small in size (e.g. less than 1,000 lines of code, not including comments & integration tests). Exceptions may be made if previously agreed upon.
• My PR passes Checkstyle validation based on the Code Style Guide.
• My PR includes Javadoc for all new (or modified) public methods and classes. It also includes Javadoc for large or complex private methods.
• My PR passes all tests and includes new/updated Unit or Integration Tests based on the Code Testing Guide.
• If my PR modifies the REST API, I've linked to the REST Contract page (or open PR) related to this change.